Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace GetOnlineFeaturesResponse with GetOnlineFeaturesResponseV2 in… #2214

Merged
merged 5 commits into from
Jan 14, 2022

Conversation

tsotnet
Copy link
Collaborator

@tsotnet tsotnet commented Jan 14, 2022

… Python

Signed-off-by: Tsotne Tabidze tsotne@tecton.ai

What this PR does / why we need it: In Java server we started using GetOnlineFeaturesResponseV2 instead of GetOnlineFeaturesResponse as a response proto type. This PR changes Python codebase to also use GetOnlineFeaturesResponseV2. We also get rid of the previous GetOnlineFeaturesResponse definition completely and rename GetOnlineFeaturesResponseV2 to GetOnlineFeaturesResponse. This is a backwards compatible change for the SDK. Namely, we aren't changing the results of feature_store.get_online_features(...).{to_dict(),to_df()} calls. However, the Python HTTP server's response will be different. This is an example of a response:

{
  "metadata": {
    "feature_names": [
      "driver_id",
      "conv_rate",
      "acc_rate",
      "avg_daily_trips",
      "conv_rate_plus_val1",
      "conv_rate_plus_val2"
    ]
  },
  "results": [
    {
      "values": [
        1001,
        0.7037263512611389,
        0.8724706768989563,
        308,
        1.703726351261139,
        10.703726351261139
      ],
      "statuses": [
        "PRESENT",
        "PRESENT",
        "PRESENT",
        "PRESENT",
        "PRESENT",
        "PRESENT"
      ],
      "event_timestamps": [
        "1970-01-01T00:00:00Z",
        "2021-12-31T23:00:00Z",
        "2021-12-31T23:00:00Z",
        "2021-12-31T23:00:00Z",
        "1970-01-01T00:00:00Z",
        "1970-01-01T00:00:00Z"
      ]
    },
    {
      "values": [
        1002,
        0.038169607520103455,
        0.48534533381462097,
        332,
        2.0381696075201035,
        20.038169607520103
      ],
      "statuses": [
        "PRESENT",
        "PRESENT",
        "PRESENT",
        "PRESENT",
        "PRESENT",
        "PRESENT"
      ],
      "event_timestamps": [
        "1970-01-01T00:00:00Z",
        "2021-12-31T23:00:00Z",
        "2021-12-31T23:00:00Z",
        "2021-12-31T23:00:00Z",
        "1970-01-01T00:00:00Z",
        "1970-01-01T00:00:00Z"
      ]
    },
    {
      "values": [
        1003,
        0.9665873050689697,
        0.7793770432472229,
        779,
        3.9665873050689697,
        30.96658730506897
      ],
      "statuses": [
        "PRESENT",
        "PRESENT",
        "PRESENT",
        "PRESENT",
        "PRESENT",
        "PRESENT"
      ],
      "event_timestamps": [
        "1970-01-01T00:00:00Z",
        "2021-12-31T23:00:00Z",
        "2021-12-31T23:00:00Z",
        "2021-12-31T23:00:00Z",
        "1970-01-01T00:00:00Z",
        "1970-01-01T00:00:00Z"
      ]
    },
    {
      "values": [
        0,
        null,
        null,
        null,
        null,
        null
      ],
      "statuses": [
        "PRESENT",
        "NOT_FOUND",
        "NOT_FOUND",
        "NOT_FOUND",
        "PRESENT",
        "PRESENT"
      ],
      "event_timestamps": [
        "1970-01-01T00:00:00Z",
        "1970-01-01T00:00:00Z",
        "1970-01-01T00:00:00Z",
        "1970-01-01T00:00:00Z",
        "1970-01-01T00:00:00Z",
        "1970-01-01T00:00:00Z"
      ]
    }
  ]
}

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Use a new response schema for online feature retrieval in Python

… Python

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pyalex, tsotnet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
@codecov-commenter
Copy link

codecov-commenter commented Jan 14, 2022

Codecov Report

Merging #2214 (eedac3c) into master (2a95629) will increase coverage by 0.14%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2214      +/-   ##
==========================================
+ Coverage   84.98%   85.12%   +0.14%     
==========================================
  Files         105      105              
  Lines        8450     8453       +3     
==========================================
+ Hits         7181     7196      +15     
+ Misses       1269     1257      -12     
Flag Coverage Δ
integrationtests 73.23% <76.08%> (-0.13%) ⬇️
unittests 59.83% <68.47%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...n/tests/integration/online_store/test_e2e_local.py 100.00% <ø> (ø)
sdk/python/feast/feature_store.py 91.82% <100.00%> (+0.23%) ⬆️
sdk/python/feast/infra/online_stores/dynamodb.py 85.61% <100.00%> (ø)
sdk/python/feast/online_response.py 100.00% <100.00%> (+5.26%) ⬆️
.../integration/online_store/test_universal_online.py 98.13% <100.00%> (ø)
sdk/python/tests/unit/test_proto_json.py 100.00% <100.00%> (ø)
sdk/python/feast/proto_json.py 92.42% <0.00%> (-3.04%) ⬇️
sdk/python/feast/infra/online_stores/datastore.py 89.15% <0.00%> (+1.88%) ⬆️
sdk/python/feast/type_map.py 66.40% <0.00%> (+6.25%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a95629...eedac3c. Read the comment docs.

Tsotne Tabidze added 2 commits January 14, 2022 11:19
Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
for row_idx, result_row in enumerate(online_features_response.results):
result_row.values.append(values[row_idx])
result_row.statuses.append(FieldStatus.PRESENT)
result_row.event_timestamps.append(Timestamp())
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So python feature server won't provide real event timestamp? and correct status as well

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, this is entities. I got it.
But why we put entities in feature vectors?

Copy link
Collaborator Author

@tsotnet tsotnet Jan 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this note got lost during the conflict resolution:

# Note: We temporarily store entities in the same field as features for backwards compatibility.
# Java server uses the same GetOnlineFeaturesResponse but does not return entities right now.
# We need to unify the logic between Python & Java on this.

Basically Python SDK used to return entities as part of the online features response and I don't want to break that with this PR. Although we should have a followup conversation about how to properly resolve this.

):
# Add more feature values to the existing result rows for the request data features
for feature_name, feature_values in request_data_features.items():
proto_values = python_values_to_proto_values(
feature_values, ValueType.UNKNOWN
)

online_features_response.metadata.feature_names.val.append(feature_name)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we put values from request back into response? It seems inconsistent with the fact that we don't put entities in response

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same reason as for entities

@pyalex
Copy link
Collaborator

pyalex commented Jan 14, 2022

/lgtm

@feast-ci-bot feast-ci-bot merged commit 1b98ec9 into feast-dev:master Jan 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants